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09/771,340, filed Jamiar>' 26, 2001, and 09/503,881, filed February 14, 2000f £B Q 7 Z002 • 



Filed of the Invention 
[0003] The present invention relates to steganography and, more particularly, to the 
detection of digital watermarks in media such as images, video and audio signals. 



|0004] The technology for embedding digital watermarks in images is well known. 
Likewise, the technology for detecting and reading the data payload carried by digital 
watermarks is well known. Assignee's U.S. Patent Application No. 09/503,881 , filed 
February 14, 2000, and U.S. Patent Nos. 5,862,260 and 6,122,403 illustrate examples of 
various watermarking techniques. Artisans in the field know even more. Commercial 
systems are available for performing such operations. 

[0005] Many watermarking systems redundantly embed the same watermark data in 
multiple regions of an image. Ofien watermarking systems embed data in images in a 
perceptually adaptive maimer. That is, the amount of watermark, signal in each region of 
an image is adjusted in accordance with the characteristics of the image in the particular 
region. The watermark may even be absent in some regions of the image. The purpose 
of so adjusting the watermark signal is to insure that the watermark signal will not be 
visible to an ordinary viewer of the image. Since the strength of the Avatennark signal 
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varies from region to region, the signal is more easily detected in some regions of an 
image than in other regions of the image. 

|0006) Systems for delecting watermarks generally sequentially examine the various 
regions of an image, seeking to detect the waterqj^rk. Generally, the amount of 
computational resources available is limited and if a watermark is not detected in a region 
as a result of applying a certain amount of computational effort, the detection operation 
moves on to the next region of the image and the process is repeated. 

Summary of the Invention 
[0007] The present invention enables detection of the presence of a watermark in an 
■ efficient manner. One embodiment involves a multi-step process. First, the image is 
examined to determine which regions of the image have characteristics such that there is 
a high probability that a watermark sigjial can be detected in that region of the image. 
Next the regions that have a high probability that a watermark can be detected (in 
contrast to all regions of the image) are examined to find watermark data. ]n order to 
determine the probability of finding watermark data in a particular region of an image, 
the amount of 'Variance" in the intensity of the pixels in the region is examined. For 
example a region that is entirely white or entirely black has zero variance. Such a region 
cannot carry watermark data: hence regions with zero or low variance can be eliminated 
from further processing. Furthermore, if high variance in a region is a result of the fact 
that the region has an abrupt border or edge between two highly contrasting regions, the 
high variance does not necessarily indicate a high probability that a watermark signal will 
be detected in the region. Therefore, after regions with high variance are located, these 
regions are next examined to look for regions with edges between areas of different 
luminance, w^hich are spread over the entire region. The regions with the high variance 
and with edginess that is sprea.d widely in the region are selected for further processing to 
detect watermark data. In another embodiment, however, regions with high variance are 
not always indicative of a high detection probability. 
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[0008] For those regions selected for further processing, the detection process can be 
enhanced by filtering the data prior to appl^ong a watermark detection program so as to 
increase the signal to noise ratio of the watennark signal. First a high pass filter (e.g. a 
Laplacian operator) is applied to each region. This filtering operation in effect 
establishes a new intensity value for each pixel in the region. >lexl a nonlinear operator 
(e.g. a signum function) is applied to the output from the first filler operation. The 
resulting data in each region is then processed in a normal maruier to detect watermark 
data. ! 

|0009] In other embodiments, additional probability factors, or region selection criteria, 
are used to identify image regions having a high probability of containing watermark data 
therein. 

[0010] The foregoing and other features and advantages of the present invention will be 
more readily apparent from the following detailed description, which proceeds with 
reference to the accompanying drawings. 

Brief Description of the Drawings 
JOOll] Figure 1 shows an image with different regions. 

{0012) Figure 2 shows the Figure 1 image divided into regions for processing. 

[0013] Figure 3 illustrates the pixels in different regions of an image. 

[0014] Figure 4 shows a flow diagram for one embodiment of the present invention. 

|0015] Figure 5 shows a floAv diagram for additional steps that.can be used. 

[001 6J Figure 6 shows a system diagram for practicing an embodiment of the present 
invention. 
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10017] Figure 7 is a graph showing a relative probability of a successful watermark 
detection for a given area having a particular variance. 

1 001 8J Figure 8 shows an edginess detection method in relation to an image portion. 

f 

|0019] Figures 9a-9c show another edginess detection method in relation to an image 
portion. 

10020] Figures 1 Oa-lOc show an image portion that is divided by regions for processing. 

[0021] Figure 1 1 shows city-block and diagonal distances between centers of detection 
blocks. 

[0022] Figure 12 shows a keep away zone near a border of an image. 

10023] Figure 13 shows a neighborhood of detection blocks. 

|0024) Figure 14 illustrates a system diagram for practicing an embodiment of the 
present invention. 

Detailed Description 

|0025] Digital watermarks are generally inserted into images in a redundant manner. 
That is, images are divided into regions and the same digital watennark data is inserted 
into each region of the image. The ability of a particular region of an image to effectively 
cairy digital watermark data depends upon the characteristics of the image in the 
particular region. Different areas in an image may have more or less ability to carry 
watermark data. For example an area in an image that is entirely white or entirely black 
will not have the ability to carry watermark data without changing the appearance of the 
area. Modem watermarking programs use visually perceptual adaptive techniques when 
inserting watennark data into an image. The amount of watermark energy inserted into a 
region is adjusted depending on the characteristics of the region so as to avoid changing 
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the visual appearance of the image. For example, no watennark energy would be applied 
to an area of an image that is entirely white or entirely black. 

[0026] Watermark detection programs generally divide an image into regions and then 
sequentially try to read watemiark data frorn each of the regions in the image. Generally 
several attempts are made to detect watermark data in each region of an image. This is a 
computationally costly endeavor. i 

10027] The present invention shortens the processing time and reduces the 
computational power required to find a watermark in an image by first identifying those 
regions of the image that have a high probability that a watermark can be detected in the 
region. Then, regions with high probability rather than all regions are examined to locate 
watermark data. 

[0028] It is noted that there are a number of different probability factors that can be 
considered in connection with watermark detection. For example, one can consider the 
probability that data found by a watermark detection program is in fact identical to ihe 
data that was inserted by the program that inserted the watermark. The probability 
discussed herein is different. The probability factors discussed herein relative to the 
present invention relates to the probability that a region of an image with certain 
characteristics can in fact be carrying watermark data. 

[0029] Figure 1 illustrates an image 2, which has a number of different identified 
regions. Regions with various types of specific characteristics have been shown in order 
to illustrate the invention. Naturally in most images the regions would not be as 
pronounced as those shown in Figure 1 and there would be a variety of types of regions 
over the entire image 2. The present invention is applicable to any t\pe of image. The 
special image shown in Figure 1 is selected only as an example to illustrate the principles 
of the invention in an easily illustrated manner. 
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10030] In the image 2 shown in Figure 1 , region 1 0 is entirely while, region 1 1 is 
entirely black and in region 12, the pixels of the image have a variety of luminance 
values. If a perceptually adaptive watermarking program were used to insert watermark 
data in an image such as image 2, no watermark data would be inserted in regions 10, 11. 
Thus, a program, which tried to detect watermark«data in regions 1 0, and 1 1 , would spend 
time examining these regions, but it would find no watermark data. 

|003I] Figure 2 shows the image 2 divided into regions. These regions can also be 
referred to as detection blocks. In order to detect digital watermark data, a typical 
watermark detection program would process the regions of an image (such as those 
regions shown in Figure 2) in some sequential order. Each region would be examined to 
determine if watermark data could be detected. Such examination requires a significant 
amount of time and'or computational resources. In some applications time and/or 
computational resources are limited. 

|0032] The present invention provides a way to pre-process or filter an image to 
determine the regions that are most likely to contain watermark data. The initial 
processing of each region, that is, the initial filtering, is done very quickly and the 
regions, which have the most probability of yielding watermark data, are selected for 
further processing to actually detect the watermark data. That is, the time consuming 
watermark detection algorithms are only applied to the regions, which have a higher 
probability of providing watermark data. For images that are scanned at a relatively high 
resolution (e.g., 600 ppi) the present invention optionally can use only part of the image 
data in order to speedup processing. For example, high-resolution data can be down- 
sampled (e.g.,: either directly or after applying antialiasing filters) to a lower resolution 

i 

for analysis. 

10033) Figure 3 illustrates pixels in an image. It should be noted that for convenience of 
illustration, only a limited number of pixels are shown in Figure 3. The 4X4 blocks are 
shown for convenience of illustration. Of course the blocks can range in size firom 4 to 
500 pixels by 4 to 500 pixels, or more. Furthermore for convenience of illustration no 
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attempt has been made to make the locations or size of the regions in Figure 3 correspond 
to the regions in Figure 1 . In t>pical applications images are scaimed at resolutions 
higher than 75 pixels per inch (resolutions of 300, 600 and 1200 pixels per inch are 
common) and the regions examined by watermarking programs would generally cover 
many more pixels than the regions shown in Figure 3. However, the limited number of 
pixels shown in Figure 3 is sufficient to explain the principles of the present invention. 

10034] In area A of Figure 3 all of the pixels have a luminance value of zero. This area 
corresponds to an area such as area row c column 3 in Figure 2 where the entire region is 
white. In area B all the pixels have a luminance value of 9. Area B corresponds to an 
area such as the area in roAv c column 7 in Figure 2 where all of the pixels are black. In 
area C the luminance value per pixel varies between 0 and 9. Area C corresponds to an 
area such as the area in row g column 7 in Figure 2 where the pixels have a range of 
luminance. Since the pixels in area A all have a luminance of 0, there is no possibility 
that this region contains watermark data. Likewise, since all the pixels in region B have a 
luminance value of 9, there is no possibility that region B contains watermark data. The 
pixels in region C have a variety of luminance values: hence, there is a possibility that 
this region does contain watermark data. The present invention is directed to detecting 
the area of an image where there is sufficient variance in the luminance of the pixels in 
the region that the region could contain watermark data. In one embodiment, an 
*'edginess'' factor (discussed below) can be used to select between regions . that have the 
same or similar variance. In such a case, a region having a higher edginess factor is 
selected over a region with a lower edginess factor, when their variance is equal. 

10035) In one embodiment of the present invention the detection operation proceeds in 
accordance with the steps shown in Figure 4. First as indicated by block 21 the image 
being examined is scanned to detect the luminance of the pixelson the image. Next the 
pixels are broken into regions. For example each region can be square and have in the 
order of 1 0000 to 40000 pixels (that is, in the order of 1 00 to 200 pixels square). The 
exact number of pixels in each region depends on the characteristics of the particular 
detection program used. There is, however, a general advantage of using smaller regions 
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(e.g., 8X8 through 64 X 64) to calculate variance. N«imely. a smaller region is less 
likely to be affected by image rotation. There is a tradeoff for selecting a smaller region, 
however, since the variance estimate is less statistically reliable due to the smaller 
number of pixel samples. As indicated by block 22, the variance in the luminance of the 
pixels in each block is calculated. The followingoformula is preferably used: 

Variance = s\im ((intensity/ / (number of pixels)) - (mean intensity)^ 

[0036] If the variance is less than a specified threshold the region is eliminated from 
further consideration. The threshold value selected will depend upon the size of the 
regions into which the detection program divides the image and upon the characteristics 
of the watermark as measured over a representative set of images. However, for a typical 
image with a program that divides the image into regions, which are in the range of about 
1 00,000 to 300,000 pixels, the value can be in a range of 100 to 500. Of course the pixel 
riange can be smaller if a lower resolution (e.g., 100 dpi) image (or image area) is 
evaluated. 

I0037J An optimal minimum variance threshold is found to vary with resolution. That 
is, the higher the resolution, the higher the minimum variance should be. This is 
particularly the case when high-resolution data is efficiently doAvn-sampled, e.g., without 
using antialiasing filters. Table 1 shows a relationship between optimal minimum 
variance thresholds and resolution. Of course, these minimum values may vary 
depending on image characteristics, scanner error, precision vs. efficiency requirements, 
etc. For instance, these minimum values may decrease depending on the above 
considerations. 





Resolution (dpi) 


75 


100 


150 


300 


600 


Minimum variance 


50 


66 


100 


200 


300 



Table 1 : Minimum variance at different resolutions for optimal results 
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10038) Variance of pixels in a region tends to increase with resolution. This is 
particularly tnie at higher resolutions where nearest neighbor down sampled data (which 
may be highly aliased) is used to calculate variance. Increasing the variance threshold 
with resolution prevents selection of blocks with spurious variance caused by borders, 
paper texture, noise etc. 

10039] Another variance determination method relies on a distribution formed by 
gathering a statistically significant amount of variance data across a broad range of 
images. Separate distributions dl and d2 in Figure 7 are computed for regions that have a 
higli likelihood of successfully detecting a watennark and for regions that have a low 
likelihood of successfully detecting a watermark, respectively. A probability value 
associated with a variance for a detection block, e.g., a probability value indicating a 
likelihood of finding a watermark signal in. a particular region having a given variance 
value, can then be determined for any given variance value. Thresholds can either be 
determined empirically, e.g., through Bayes' Rule or other hypothesis tests. This 
probability value is compared against a tlireshold or a set of thresholds to decide whether 
to keep the particular variance block. A look up table or software algorithm is preferably 
used to implement the distribution shown in Fig. 7. Note that the distributions shown in 
Fig. 7 are for illustrative purposes only. Indeed, the actual distribution could be different, 
e.g., multi-modal, non-Gaussian or a mixture of Gaussians. Also, the principles 
discussed with respect to Fig. 7 can be extended to other metrics as well (e.g., variance 
and edges) to form multivariate distributions. 

10040] To create the distributions shown in Fig. 7, where a probability of finding a 
watermark signal is graphed in relation to variance, a statistically significant number of 
variance values are determined from a respective number of sampled variance detection 
blocks. Each of the sampled, variance detection blocks is read to determine whether it 
contains a watermark signal. This detection data is used to generate the probability 
distribution curv'es for given variances. 
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10041] While there is a low probability that areas with a very low variation in luminance 
contain watennark data, there is also a low probability that certain areas, which have a 
very high variance in luminance, contain watermark data. For example, the area in row c 
column 6 contains the border between black area 1 1 and the remainder of the image. In 
areas such as the area at row c column 6, the variance in luminance would be higli due to - 
the edge effect; however, the high variance in luminance in an area such as row c column 
6 would not indicate a high probability of finding watermark data. In a region such as 
row c column 6 the **edginess spread" is low. ]f a region has a low ''edginess spread", the 
probability of finding watennark data is relatively low. 

|0042] Thus, after the regions with high luminance variation values are found, those 
regions are tested to determine "edginess spread". That is, to locate regions where the 
variance is concentrated along a division between regions each of which have a low 
variance. Regions where variance in luminance is concentrated along a division between 
regions, each of which has a low variance in luminance, are said to have a low edginess 
spread. 

I0043I In one embodiment, edginess is found by filtering the data with an edge operator 
such as a Laplacian operator or filter, which examines the pixels suironnding each pixel 
to calculate a New Pixel Intensity value (designated NPl value) and edginess spread 
value (ES) according to the following equations: 

NPl = Abs Value (4 x Intensity - (sum of intensities of pixels above, below, right 
and left)), where "Abs Value" means "take Absolute value of. 

i 

Calculate an NNPl value for each pixel as follows: 
NNPl = 1 if NPl exceeds a Tl : and 

= 0 if NPl is less than or equal to Tl . 

ES = (Sum of NNPl for all pixels) / total number of pixels, where Tl is a 
"threshold" with a value selected to be near the average value of NPL 
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The above calculation gives a second value (ES or edginess spread) for each region. The 
luminance variance value and the edginess-spread value are then combined to give a 
''probability index'' which indicates the probability of finding a watermark in a particular 
region. Alternatively, a difference operator (e.g., a Sobel operator, etc.) could be used to 
account for both variance and edginess. 

|0044] In another embodiment, edginess is determined by evaluating some of a pixel's 
(or area's) neighbors in comparison to that pixel. For example, a difference in graylevels 
(or color data) between neighboring pixels is compared to determine an edge or edginess 
value. With reference to Figure 8, area x is compared to its horizontal (h) neighbor and 
vertical neighbor (v) to determine an edginess count. For a comparison with horizontal 
neighbor h, an edginess count is preferably incremented when: 

|x - h I > Te , where Te is an edginess threshold, and x and h are a 
measure of their respective pixel (or area) graylevel. 

Similarly, for a comparison with vertical neighbor v, the edginess count is incremented 
when: 

|x-v| > Te , where V is also a measure of its respective pixel graylevel- 

Tbis process can be repeated for some or all of the areas within the edginess 
determination block 20. When area x is positioned at a boundary (e.g., pixel a) of block 
20, the neighboring h pixel is preferably zero (0). Alternatively, a pixel value outside of 
block 20 that is located in the horizontal position h is used. 

*** • t 

|0045) The total edginess count for block 20 can be compared against a predetennmed 

number to determine whether to further use block 20 in the watermark detection process. 

Or the edginess count can be used to rank various edginess determination blocks. Of 

course this process can be modified without deviating from the scope of our invention. 
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For example, instead of sampling a left horizontal neighbor, a right horizontal neighbor 
can be sampled. And instead of looking down to the vertical neighbor, a neighbor above 
can be sampled. In another case, a pixel x is compared to several horizontal neighbors 
and to several vertical neighbors, or even diagonal neighbors. Also, the illustrated 
edginess detection block 20 need not be limited t^ a 3 X 3 area as shoAvn. Indeed, the 
block area can be increased (e.g., to an 8 X 8 through 64 X 64 area). 

|0046] This process can be repeated for some or all blocks through out an image. 

10047] There are many factors to consider when determining an edginess threshold 
value. Since the edginess factor helps determine where the variance is coming from, a 
low edge count may indicate that the variance is confined to a small image region. In 
contrast, a large edge count may indicate that variance is distributed throughout an image 
region. A lenient threshold, e.g., 0-2 (or a difference of 0-2 graylevels between adjacent 
pixels to constitute an edge), will allow influence from random noise or from small image 
variations. A larger edginess threshold (e.g., 2-8) may include influence from a 
watermark signal. Increasing the edginess threshold may also reduce sensitivity to 
spurious edges caused by borders, paper texture, scanner backgrounds and noise. There 
is a tradeoff, however, since a larger threshold may miss a watermark signal embedded at 
a low strength. These same factors can be considered to determine an appropriate 
edginess count threshold. 

[0048] In another embodiment, the edge threshold is resolution dependent, meaning the 
edge tlueshold changes are based on sample resolution. In still another embodiment, an 
edginess threshold is determined based on image characteristics. In this case, the 
edginess threshold adapts to the image (or scanner) characteristics. In yet another 
embodiment, the variance and/or edge threshold is adaptively detemiined by the size of 
the image or the available processing power/memory. 

[0049] Another edginess method that is particularly useful to detect diagonal edges is 
now discussed. A horizontal map and a vertical map are detemnined based on pixel 
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values in a edginess detection block. These maps are generated by determining those 
areas (or pixels) that have sufficient differences in graylevels when compared to 
neighboring pixels. The horizontal map is constructed using the horizontal techniques 
discussed above with respect to Figure 8. High graylevel difference areas are designated 
as 1 (see Fig. 9a). A vertical map is conslnicted using the same vertical tecliniques as 
discussed above with respect to Figure 8. High graylevel difference areas are designated 
as ] (see Fig. 9b)- The horizontal and vertical maps are then combined (e.g., with a 
Boolean **0R" operation or other combination technique) on a per pixel; basis. The 
resulting map is used as the edge map (Fig. 9c). The edginess count of the new map (Fig. 
9c) is counted to detemiine a total edginess count for the edginess detection area. The 
edginess counts obtained by this method are more robust with respect to distortions 
caused by operation such as image rotation. 

|0050] The luminance variance value and the edginess-spread value can be combined in 
a number of ways to obtain a numeric probability index that a region can contain 
watermark data. For example the values can be combined as follows: 

Probability index = ((variance value) /1 00) 10 (edginess value) 

|0051] Table 2 is an example of a probability index, which results from a number of 
different values of luminance variation, and a number of values of edginess spread. 



Variance 


Edginess 


Probability 


' value 


value 


Index 


300 


7 


10 


500 


2 


7 


700 


9 *" 


16 



Table 2: Probability Index 
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[0052] In the above example, the region with the probability index of 1 6 would be 
examined first, followed by the region with an index of 10. Regions with an index value 
of less than 10 would only be examined if the other regions that are examined do not 
result in the detection of watermark data of sufficient reliability, 

|0053] It is noted that the equation for combining the values of luminance variation and 
edginess to obtain the probability index for a region was determined empirically. The 
equation given above does not take into account the magnitude of the change in 
himinance across an edge. The following equation for calculating edginess spread takes 
into account the magnitude of the change in luminance across an edge. 

ES = (Sum of NPl for all pixels that exceed Tl/ total number of pixels). 

By testing the success obtained with different groups of images of interest which have 
different characteristics one can determine which equation gives the best results for 
images with a particular set of characteristics. 

10054] In other embodiments, we do not combine the edginess and variance factors in 
the manner discussed above. Instead, detection blocks are selected if they meet both the 
threshold edginess and/or variance factors. Or variance and edginess may be used 
together or separately and/or in combination with the other probability factors discussed 
herein. 

I0055J It is noted that U.S. Patent Application No. 09/074,034 filed iMay 6, 1998, 
entitled "Methods and Systems for Watermark Processing of Line Art Images," describes 

i 

a technique for inserting watermarks into a lined image by varying the width of the lines 
to indicate watermark data. The present invention would still produce satisfactory results 
with watermarks of the type described in the above referenced application. The reason is 
that the line widths in a typical image, which uses the technique described in the above 
application, have a width significantly smaller than the size of a pixel in an image from a 
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t>'pical 300 or 600 DPI scanner The edginess measureinent detected by the present 
invention relates to edges between regions, each of which are wider than a single pi>cel. 

10056] The present invention can optionally utilize additional filtering to enhance the - 
possibility of finding watermark data in the regions selected fox fiirther processing by the 
above-described teclinique. A flow diagram showing how the additional filtering is 
performed is shown in Figure 5. The additional steps shown in Figure 5 facilitate the 

i 

detection of watermark data in those regions selected /or further processing by the steps 
shown in Figure 4. 

|0057] ]n the process shown in Figure 5, regions that have a high probability of carrying 
watermark data are selected for further processing as described above. HoAvever with the 
steps shown in Figure 5, the regions selected for further processing are filtered prior to 
the detection step in order to enhance the detection process. The filtering enhances the 
probability that watermark data (if present) will be detected when a region is later 
processed in a normal or conventional manner to find a watermark. The filtering is done 
in two steps. First as indicated by block 52, a high pass filter (e.g. a Laplacian operator) 
is applied to the data. Next as indicated by block 55 a non-linear operator (e.g. signum 
function) is applied to ihe filtered data. Finally the data is processed in a conventional 
manner to detect the watennark data. 

100581 The first step passes the data from a region through a filter with high pass or 
edge detection characteristics. For example a Laplacian (or Sobel or Roberts, etc) 
operator can be applied to each block that was selected for further processing. In the 
specific embodiment shown here, the high pass filter computes a new intensity value at 
each pixel in the block as follows: 

Filtered intensity = (Old intensity) - (average intensity of the 8 neighbors of the 
pixel) 



-15- 



SWSrlmp 8/31/2001 P0419 



EXPRESS MAIL EL818267258L'S 



10059] The second step applies a nonlinear operator (e.g., a signum operator etc) to the 
filtered output of the high pass or edge detection filter. The filtered intensity (Fl) of each 
pixel calculated as given above is modified as follows: 

New Intensity = a if(Fl>Tl); ' s 
= b if (T2<=Fl<=Tl); and 
= cif(Fl<T2), 

where: a, b, and c are values, and Tl and T2 are thresholds selected to implement a 
specific nonlinear operator. 

10060] In the specific embodiment shown herein a signum function is used to calculate 
a new intensity for each pixel according to the following equation: 

New intensity = 1 if (Filtered intensity > 0) 
= 0 if (Filtered intensity = 0) 
= -1 if (Filtered intensity < 0) 

|006] ] Tlie high pass filter attenuates the low frequencies and amplifies the contribution 
from the higher frequencies in each block. The contribution to the low frequencies is 
mostly from the host image content. Higher frequencies from the watemiark signal are 
amplified. The nonlinear operation in effect whitens the noise caused by the host image 
content in the frequency domain, increasing the signal-to-noise ratio of the watermark 
signal. 

10062] It is noted as described above, a tw^o-dimensional high pass filter is first applied 
to the data and then the non-linear operator is applied to the result. With many types of 
images better detection can be achieved by applying a one dimensional high pass filter in 
the horizontal direction, applying the non linear operator to that result, applying a one 
dimensional high pass filter in the vertical direction, applying the non linear operator to 
that result, and then summing the Iavo partial results. With other types of images better 
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results can be achieved by appl>nng the one-dimensional high pass filters in varions other 
directions. 

10063] Since some watermarking programs redimdantly embed watennark data in 
multiple blocks in an image, in order to further enhance the ability to detect the 
watennark data from such programs the following technique can be used. Following the 
non-linear filtering operation, the power spectrum of several blocks canibe added 
together. Due to the redundant embedding, the watennark frequencies repeat through 
several blocks, the power at those frequencies adds up if the power spectrum of several 
blocks is added together. The image frequencies from block to block are generally non- 
repetitive and hence they get averaged out when the power spectrum of several blocks are 
added together. The power spectnim that results from adding together the power 
spectnim from several blocks contains a higher signal-to-noise ratio watermark signal. 
The power spectrum can then be more easily conelated with the power spectrum of the 
watermark- 

10064] A system for practicing one embodiment of the present invention is shown in 
Figure 6. The system includes a conventional computer 60 with an associated display 61, 
an associated document scanner 62 and an associated printer 63. The computer 60, 
display 61, scanner 62 and printer 63 are conventional components of personal computer 
systems such as those marketed by vendors such Compact Computer Company, Dell 
Computer Company, Gateway Computer Corp. etc. 

10065] One embodiment of the present invention is practiced under control of programs 
A, B and C, which are stored in computer 60. Program A is a conventional watermark 
detection program. Program A processes regions of an image to locate watermark data 
after program B selects the regions of the image which should be processed and program 
C filters the data from such regions. 

10066] Programs which process the pixels in an image to locate watermark data are 
included in such commercially available programs as the program entitled "Photoshop" 
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which is marketed by Adobe Corporalion or the program "Corell DRAW which is 
marketed by Corel Corporation, or the program "Micrografx Picture Publisher' which is 
marketed by Micrografx Corporation. Such programs divide an image into regions and 
process each region in order to detect the presence of watermark data. With the present 
invention the same mechanism is used to process^fhe data from each region of an image; 
however, all the regions of an image are not processed in order. 

10067] Program B selects regions of an image, which have a high probability of 
containing watermark data by first selecting regions that have a high variation in 
luminance, and a high amount of edginess spread as previously described. Program C 
filters the regions selected for further processing using the two steps process previously 
described. 

10068] In the embodiment of the invention described above, program 51 (shown in 
Figure 5) which selects blocks for further processing merely indicates to the subsequent 
filtering program which blocks should be processed further. The block selection program 
could be used to acquire other infonnation about the various blocks in the image. Such 
additional information could be passed to the filtering programs shown in block 53 and 
55 and to the watermark detection program indicated by block 57 to quickly tune these 
programs to the characteristics of the image in particular regions. 

10069] The present invention includes a wide range of additional probability factors. A 
probability factor can be viewed as a selection criteria or rule that is used to identify those 
regions in an image which have.a high likelihood of including a watermark signal. These 
image regions generally include image characteristics that are conducive to (or indicative 
of) hiding or carrying a watermark signal. Or these image regions may be located in a 
particular advantageous area, or may include significant signal strength. Probability 
factors are used to select a plurality of detection blocks, which are image regions 
identified as having a relatively high probability of including a Avatermark signal. 
Variance and edginess are just a few of our inventive probability factors. There are many 
more. 
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j0070] Consider an embodiment in which detection blocks (or areas) float, instead of 
being sequentially anranged as in Figure 2. Allowing detection blocks to float to various 
image regions and, optionally, to overlap with other detection blocks, allows for 
improved detection of off-centered watermarks. Moreover centering a detection block on 
an image region, which includes characteristics that may indicate a region of high 
detection probability, can help to reduce watermark signal estimation eijor -- such as 
rotation and scale error — particularly if a captured region is approximately centered m a 
floating detection block. A floating detection block is illustrated with reference to an 
image (or image portion) 30 shown in Figures 10a and 10b. In Figure 10a image 30 is 
sequentially segmented into detection regions (e.g., a, b, c and d). For this example 
region 31 is assumed to include characteristics indicating a high probability of containing 
a watermark signal. Region 31 is off centered with respect to the sequential detection 
blocks a, b, c and d shown in Figure 10a. Accordingly, a watermark detector may not 
successfully detect the presence of a watermark signal. Detection chances are improved 
if a detection block 32 (Figure 1 Ob) is allowed to float in order to enclose a larger portion 
of region 31 . Centering floating detection block 32 on region 31 allows for a higher 
probability of detection and lowers watemiark signal rotation estimation error. Although 
Fig. 1 Ob encloses the entire region 31, it may not always be possible to do so, depending 
on a floating detection block size. 

|D07] ] Figure 1 Oc shows a plurality of floating detection blocks, illustrated by dashed 
lines, which are arranged over an image or image portion 34. Preferably, a floating 
detection block is positioned in a region that has high probability characteristics, e.g., 
such as having adequate variance and edginess or based on other probability factors 
discussed herein. A floating block can be centered on or otherwise positioned around 
such a high probability region. In one embodiment, a detection^block covers a larger 
region of the image than does the respective blocks used to determine variance and 
edginess. A variance block size may also be larger than an edginess block size, or vice 
versa. 
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10072] 111 order to increase the effectiveness of a plurality of floating detection blocks, 
additional probability factors can be \ised to arrange or position the blocks over an image. 
Since these probability factors often involve a compromise between processing efficiency 
and memory considerations, a fixed number of detection blocks can be selected in some 
embodiments. The fixed number of detection blocks can be divided into subsets. For 
example, a first subset of detections blocks can be processed according to probability 
factors that maximize the detection of a digital watermark synchronization or orientation 
signal. Or the first subset can be selected to identify the rotation and/or scale of a 
watermark signal. Or the first subset can be selected based solely on processing speed 
requirements. A second subset of detection blocks can be processed using different 
criteria, e.g., to maximize detection of a message payload or signal translation, or to 
balance memory constraints. 

|0073] Several competing factors are preferably balanced to achieve an optimal number 
of members for each detection subset. First is a consideration that a watermark signal 
may be embedded in the image with a low signal-to-noise ratio (SNR). A low SNR is 
sometimes used with digital watermarks to minimize visibility of an embedded 
watermark signal Second is a consideration of detection time constraints that are often 
placed to establish a maximum time to determine whether an image includes a watermark 
signal- This constraint suggests that a fewer number of blocks should be examined. In 
contrast, there is often a need to accurately detect the watermark signal, which suggests 
that more blocks should be examined. ]f time and memory limitations were not a 
concern, this later approach would almost certainly be preferable. Yet a watermark 
system designer is faced with real world constraints. Accordingly, a watermark detection 
system preferably balances such considerations when determining an optimal number of 
detection blocks, and whether to allocate such detection blocks into a first and second 
subsets. For an 8 X 1 1 inch, 100 dpi image that is segmented into 128 X 128 blocks, 
the number of detection blocks preferably falls within a range of 12-48 blocks. More 
preferably, the number of detection blocks falls within a range of 26-36 blocks. These 
blocks can be allocated into a first and second subset as mentioned above to balance 
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various system requirements. Of course, these ranges many vary depending on block 
size, resolution, image size, and image characteristics. 

(0074) In one embodiment, a first subset of detection blocks is used to determine 
whether a watermark signal is even embedded within the image, e.g., through the 
detection of a watermark component such as an orientation or s>Tichronization signal- 
The presence of a watermark component announces the presence of a Ayatermark within 
the image with a higli certainty. If no watermark component signal is found during the 
examination of the first subset, the image is preferably deemed unmarked and is likely 
rejected. As a result it is important that the first subset of detection blocks collectively 
contain enough watermark signal to be able to detect a watermark component signal, if 
present. In many watermark designs, the coverage or placement of a watermark within 
an image is small. Visibility requirements may force the digital watermark to be 
embedded in regions with diverse characteristics. Accordingly, we have found that it is 
advantageous to increase the block coverage (e.g., decrease detection block overlap for 
floating blocks) for the first subset of blocks in order to increase the chance of locating a 
watermark component. 

10075] In particular, we established a proximity metric (one of our probability factors) 
to help ensure broad coverage for the first subset of detection blocks. A minimum ''city- 
block" distance between centers of selected detection blocks is set, and is preferably in a 
range of 2-8 city-block centers. (The centers of detection blocks x and y, along with 
additional blocks, are represented by hexagon-shaped dots in Figure 1 1 . The city block 
distance between blocks x and y is 4.). Additional criteria can be set to further ensure 
broad detection block coverage in the first subset. For example, a minimum diagonal 
distance between block centers can be established- Preferably, the minimum diagonal 
distance is. in a range of 2-6 blocks. (The diagonal distance between blocks x and z is 3 
as shown in Figure 1 1 .). 

|0076] A second set of proximity metrics can be used to regulate overlap for the second 
s\ibset of detection blocks. In some embodiments it is advantageous to increase block 
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overlap in Ihe second subset to help focus Avatermnrk detection efforts on high probability 
image areas. Accordingly, the city block distance and diagonal requirements can be 
decreased. For the second subset, the minimum city-block distance behveen centers of 
selected blocks is preferably in a range of 1-4 block centers, and the minimum diagonal 
distance is in a range of 1 -3 block centers. * 

10077] Of course, for both the first and second subsets, the city block distances and 
diagonal requirements can vary depending on resolution, image characteristics, scanner 
error and characteristics, performance vs. efficiency compromises, memory requirements, 
etc. Also, instead of being measured from the center of a block, such distances can be 
measured from an edge, comer, off-center location, etc. 

|0078] In some embodiments, a detection block is segmented into subblocks, and the 
proximity metrics discussed above can be imposed on the segmented subblocks. 

10079] In one embodiment, detection blocks in the first subset are weighted according 
to their probability of including those characteristics likely to support (or hide) a 
watermark signal. Higher probability blocks are more heavily weighted. Blocks with a 
lower weighting are dropped (or conferred to secondarily) when determining the presence 
of a watermark signal. For example, consider a first subset that contains 1 0 detection 
blocks. Blocks 1-7 may collectively represent 90% of the weighting, leaving a collective 
] 0% weight for blocks 8- 1 0. Blocks 1 -7 are used as the primary detection blocks in the 
first subset, while blocks 8-10 are discarded or held in reserve. Blocks 1 -7 are then 
analyzed to detect a watermark.signal. This same type of weigliting can be applied to the 
second subset for detection of a watermark signal, hi one embodiment, the weighting is 
determined by estimating the signal-to-noise ratio in each block. This estimate is used to 
rank (or weight) the blocks. 

IQOSO] Requiring a minimum variance separation between selected detection blocks can 
be used to improve detection block selection. This probability factor forces some or all 
of the selected detection blocks to differ in variance fi-om other selected detection blocks. 
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Requiring a minimum variance separation can be a significant factor since when a large 
number of selected blocks have the same or similar variance, it often indicates that the 
selected blocks are either from the image's background or are focused in small regions of 
the image. A minimum variance separation has the effect of spreading out the blocks - 
lessening the effect of background or small region influence. Of course, a threshold can 
be selected to maximize the effect of such a minimum separation requirement. And, as 
discussed above, the variance separation threshold may be selected to vary according to 
image characteristics or resolution - creating an adaptive threshold value. 

[0081] Another probability factor establishes a "keep away" zone 36 near the borders of 
an image. (See Figure 12, in which the hashed area indicates the keep away zone 36.). 
Detection blocks preferably are not selected if centered within this keep away zone 36. 
The result is to slightly pull the block centers away from the scan borders. The 
motivation for this improvement is to reduce the sensitivity of edges caused by borders, 
scanner error, image misalignment and/or noise. Experimentally, we have found that a 
significant benefit is seldom received from blocks that are centered at an image border. 
Preferably, the keep away zone is in a range of 1 - 4 city block centers from the image 
border. Of course this distance can be expanded according to specific implementations 
and to image, scanner and/or border characteristics. 

[0082] Yet another probability factor is our ''good neighbor' rule, which is particularly 
beneficial for images at higher resolutions. The good neighbor rule ensures that 
neighboring regions also have good variance/edge characteristics so that detection block 
selection can be focused on regions that have a higher likelihood of containing a 
watermark signal. The good neighbor rule helps to prevent selection of isolated regions 
that have good variance/edge characteristics. The reasoning is that a watermark is not 
usually found in isolated regions. And even if a \yatennark is fdund, such an isolated 
region may not necessarily contribute towards successful watermark detection. 

[0083] The good neighbor rule provides that detection blocks neighboring a selected 
detection block meet established minimum variance and/or edge count requirements. 



-23- 



SWSilmp 8/31/2001 P0419 



EXPRESS MAIL EL818267258US 



Consider Fig. 13, which ilhistrates a detection block neighborhood including blocks 1-9. 
If block 5 is preliminary selected as a detection block, then a threshold number of 
neighboring blocks (blocks 1-4 and 6-9) should meet the variance and/or edge count 
requirements. These threshold values can be determined based on precision vs. 
efficiency requirements of a detection applicatiorf Moreover, isolated regions can be 
better filtered out when the threshold value is increased (e.g., all or a majority of 
neighbors meet the thresholds). Preferably, between 4-8 neiglibors must meet each of 
these edge and variance requirements before a central neighbor block is selected. Of 
course, this range can be varied according to precision required. 

1 0084) In some embodiments, an image is segmented into subblocks, which are smaller 
than the detection blocks. The good neighbor rule can be applied to these smaller blocks 
to help better filter out isolated regions of high variance and edginess. 

10085) Another probability factor helps to ensure that if a sufficient number of detection 
blocks have not been found, the variance thresholds (and optionally the proximity metrics 
discussed above) are reset to lower values and the search for acceptable blocks is 
repeated. Resetting the thresholds is particularly advantageous when an image is small 
(in which case, the city-block distance requirements discussed above may prevent further 
blocks from being selected) or when the image contrast has been reduced. 

10086) Still another probability factor relies on color saturation in a detection block. The 
color saturation level for a block is determined and then compared with a predetermined 
threshold level. If the saturation level is above the threshold, the block is selected or 
ranked. The higher the color saturation level, the higher ranking the block receives. In 
one embodiment, the saturation value is weighted (or combined) with other probability 
factors, e.g., edginess and variance. The collective metric is used to select a detection 
block. 

10087] With reference to Fig. 14, a selection module 42 implementing some or all of 
the above described probability factors is described in relation to an embodiment of a 
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watermarking detection system. An image 40 is presented for watermark detection. 
Image 40 is preferably color converted and down -samp led in modiile 41 . The color- 
converled image is then presented to selection module 42. Selection module 42 selects a 
plurality of detection blocks, which have a relatively higli probability of including a 
watermark signal embedded therein, according to some or all of the probability factors 
discussed herein. The selection module 42 generates a list of selected detection blocks 
43. The selected detection blocks 43 are processed, e.g., color converted, anti-aliased, 
and down-sampled, in processing module 44. Detection module 45 searches a first subset 
of the selected (and processed) detection blocks for a watermark component (e.g., an 
orientation signal) and/or to determine rotation, scale, differential scale, and/or shear 
from a detected watermark component. These detection results can be passed to the 
translation module 46, Translation and message detection are carried out in modules 46 
and 47, respectively, from a second subset (and optionally the first subset) of the selected 
(and processed) detection blocks, preferably only when detection module 45 detects a 
watermark component in the first subset. The first subset of blocks can be optionally 
passed to translation and message detection modules 46 and 47. 

Conclusion 

[0088] The foregoing are just exemplary implementations of the present invention. It 
will be recognized that there are a great number of variations on these basic themes. The 
foregoing illustrates but a few applications of the detailed techjiology. There are many 
others. 

|0089] ]t is noted that while the previously described embodiments discuss application 
of the present invention to images, the present invention is not so limited. Instead, the 
present invention can likewise be applied to other t>pes of media such as video and 
audio. * 

[0090] While many probability factors have been disclosed above, it should be 
appreciated that not all of these factors need to be employed in a single embodiment. 
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Instead, a selection process may only include one, several or all of the above noted 
factors. 

[0091 ) It should be appreciated that the various image blocks shown in the dra^vings are 
for illustrative purposes only. The block and im^ige sizes can be varied v^ithout deviating,, 
from the scope of the present invention. 

10092] As an alternative embodiment, all of the first and second detection block subsets 
mentioned use the same probability factors, rather than using different factors. 

[0093] To provide a comprehensive disclosure without unduly lengthening this 
specification, the above-mentioned patents and patent applications are hereby 
incorporated by reference. The particular combinations of elements and features in the 
above-detailed embodiments are exemplary only; the interchanging and substitution of 
these teachings with other teachings in this application and the incorporated-by-reference 
patents/applications are expressly contemplated. 

[0094] The above-described methods, systems and functionality can be facilitated with 
computer executable software stored on computer readable media, such as electronic 
memory circuits, RAJV], ROM, magnetic media, optical media, memory sticks, hard 
disks, removable media, etc., etc. Such software may be stored and executed on a 
general-purpose computer, or on a server for distributed use. Data structures representing 
the various luminance values, variance metrics, edginess factors, probability factors or 
methods, image signals, waterrnark signals, etc., may also be stored on such computer 
readable media. Also, instead of software, a hardware implementation, or a software- 

i 

hardware implementation can be used. 

10095) In view of the wide variety of embodiments to which the principles and features 
discussed above can be applied, it should be apparent that the detailed embodiments are 
illustrative only and should not be taken as limiting the scope of the invention. Rather, 
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we claim as our invention all such modificalions as may come within the scope and spirit 
of the following claims and equivalents iheieof 
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